fix embedding training bug for RNNLM without letter features#2586
Merged
danpovey merged 1 commit intokaldi-asr:masterfrom Aug 2, 2018
Merged
fix embedding training bug for RNNLM without letter features#2586danpovey merged 1 commit intokaldi-asr:masterfrom
danpovey merged 1 commit intokaldi-asr:masterfrom
Conversation
danpovey
reviewed
Jul 31, 2018
| if (l2_term != 0.0) { | ||
| embedding_deriv->AddToRows(l2_term, active_words, embedding_mat_); | ||
| CuMatrix<BaseFloat> l2_matrix(embedding_deriv->NumRows(), | ||
| embedding_deriv->NumCols(), kUndefined); |
Contributor
There was a problem hiding this comment.
Can you explain what the difference is, or why the old code was wrong?
Contributor
Author
There was a problem hiding this comment.
The old code does
a subset of rows of embedding_mat_ += derivs * l2_factor
But it should be
derivs += a subset of rows of embedding_mat * l2_factor
Later derivs will be added to embeeding_mat with learning rate (and possibly momentum) which makes it different.
Contributor
|
Is it possible to just use AddRows() then? I feel like there may be an
unnecessary temporary.
…On Tue, Jul 31, 2018 at 3:16 PM, hainan-xv ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/rnnlm/rnnlm-embedding-training.cc
<#2586 (comment)>:
> @@ -175,7 +175,10 @@ void RnnlmEmbeddingTrainer::Train(
if (config_.l2_regularize > 0.0) {
BaseFloat l2_term = -2 * config_.l2_regularize;
if (l2_term != 0.0) {
- embedding_deriv->AddToRows(l2_term, active_words, embedding_mat_);
+ CuMatrix<BaseFloat> l2_matrix(embedding_deriv->NumRows(),
+ embedding_deriv->NumCols(), kUndefined);
The old code does
a subset of rows of embedding_mat_ += derivs * l2_factor
But it should be
derivs += a subset of rows of embedding_mat * l2_factor
Later derivs will be added to embeeding_mat with learning rate (and
possibly momentum) which makes it different.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2586 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ADJVu7WI8ULHxvJxGuLSnp7J_F3u8SLOks5uMNc1gaJpZM4VpQeF>
.
|
hainan-xv
force-pushed
the
rnnlm-emb-bug-fix
branch
from
395f93e to
66d5feb
Compare
Contributor
|
OK, thanks. Merging. |
dpriver
pushed a commit
to dpriver/kaldi
that referenced
this pull request
Sep 13, 2018
Skaiste
pushed a commit
to Skaiste/idlak
that referenced
this pull request
Sep 26, 2018
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addressing issue #2569